js-tokens

What is js-tokens?

The js-tokens npm package is a lightweight, regex-based lexer for JavaScript tokens. It can tokenize JavaScript code without the overhead of a full parser, making it useful for tasks like syntax highlighting or extracting specific tokens from code.

What are js-tokens's main functionalities?

Tokenizing JavaScript code

This feature allows you to tokenize a string of JavaScript code into an array of token objects. Each token object describes a syntactic element of the code, such as a keyword, identifier, number, string, or operator.

const jsTokens = require('js-tokens');
const tokens = [...jsTokens('var x = 42;')];
console.log(tokens);

Other packages similar to js-tokens

acorn

Acorn is a full JavaScript parser that can parse source code into an abstract syntax tree (AST). While js-tokens only tokenizes the code, Acorn can parse it and provide more detailed information about the structure of the code.

esprima

Esprima is another JavaScript parser that can convert code into an AST. It offers similar functionality to Acorn but has different API and extension points. Compared to js-tokens, Esprima provides a more comprehensive analysis of the code.

babylon

Babylon is the parser used by Babel, and it's capable of handling modern JavaScript features. It's more feature-rich than js-tokens, offering AST generation and the ability to handle experimental syntax through plugins.

README

js-tokens

The tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.

const jsTokens = require("js-tokens");

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';

Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)

➡️ Full readme

Changelog

Version 8.0.3 (2024-02-03)

• Fixed: Extremely long string literals, template literals, regex literals, identifiers, comments, and runs of whitespace are now supported where possible. We’re talking about 10 million characters long. Previously, such long tokens could make the regex engine of the runtime give up and throw a Maximum call stack size exceeded or similar error. There are still a few [even more extreme such edge cases][known-failures], which I don’t think can be solved but are documented at least.